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(54) Picture replacement system, device and method 



(57) A picture replacement systenn for replacing a 
specified area of a present picture with a desired picture 
comprising: at least one camera; picture generation 
means for generating a desired picture for replacing; se- 
lection means for selecting a replacement area within a 
reference picture in advance and storing a position of 
the replacement area and a reference picture within the 
replacement area; detection means for detecting a re- 
placement area within a present picture taken by the 



camera based on the position of the replacement area 
and the reference picture within the replacement area; 
key signal generation means for generating a key signal 
indicating an identical part of a present picture within the 
detected replacement area and the stored reference 
picture; and replacement means for replacing an iden- 
tical part of a present picture and a reference picture 
within a part picture of the desired picture corresponding 
to the identical part based on the key signal. 
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Description 

The present invention relates to an art of replacing 
a specified area of a picture with a desired picture. The 
present invention further relates to an art of picture re- 
placement process in such a manner that the specified 
area of the present picture is naturally replaced with the 
picture showing that such picture appears to have ex- 
isted as a part of the present picture. 

Accompanied with the recent prosperity of TV 
broadcasting, a large number of advertisements have 
appeared on TV. For example, when watching a base- 
ball game on TV, viewers see not only the game played 
by the players but also advertisements shown on the 
fence of the stadium behind baseball players, for exam- 
ple, company names, logos, slogans and the like. 

There has been such circumstance that only a cer- 
tain logo is required to be replaced with another logo 
during rebroadcasting. Or the circumstance has de- 
manded to televise the specified logo at a spot thai has 
been a blank area during rebroadcasting. 

The conventional art has easily allowed a specific 
logo to replace a predetermined area of a picture. How- 
ever it has been difficult to execute such replacement 
quite naturally to show that the logo appears to have 
been a part of the picture. For example, during the ball 
game rebroadcasting, if the logo shown on the back 
fence is replaced with another logo, the ball player 
standing in front of such logo is also replaced, thus spoil- 
ing the whole broadcasting. 

An objective of the present invention is to provide 
an art of replacing a part of a present picture with a logo 
matched thereto so as to show that the picture (logo) 
appears to have existed as a part of the present picture. 

The above objective of the present invention Is 
achieved by a picture replacement system for replacing 
a specified area of a present picture with a desired pic- 
ture comprising: at least one camera; picture generation 
means for generating a desired picture for replacing: se- 
lection means for selecting a replacement area within a 
reference picture in advance and storing a position of 
the replacement area and a reference picture within the 
replacement area; detection means for detecting a re- 
placement area within a present picture taken by the 
camera based on the position of the replacement area 
and the reference picture within the replacement area; 
key signal generation means for generating a key signal 
indicating an identical part of a present picture within the 
detected replacement area and the stored reference 
picture; and replacement means for replacing an iden- 
tical part of a present picture and a reference picture 
within a part picture of the desired picture corresponding 
to the identical part based on the key signal. 

It is preferable that the camera shoots a present pic- 
ture at a fixed range for shooting a reference picture. 

It is preferable that the detection means comprises: 
area detection means for selecting a predetermined 
number of areas in a present picture, calculating a dif- 



ference between a present picture within each selected 
area and a stored reference picture and detecting an 
area presenting a minimum difference result as a re- 
placement area; and correction means for correcting a 
5 shift between the stored replacement area and the de- 
tected replacement area based on an area position of 
the detected replacement area. The replacement area 
can be detected from the present picture accurately 
even though the shooting range is shifter owing to the 
10 camera shaken in the wind. 

The above objective is further achieved by a re- 
placement means comprises a switch selectable at a 
unit of one picture element pixel based on a key signal; 
responding to the key signal indicating an identical part 
IS of a present picture and a reference picture, the switch 
selects to input a part of desired picture corresponding 
to the identical part and responding to the key signal 
indicating a different part between a present picture and 
a reference picture, the switch selects to input a part of 
present picture corresponding to the different part. 

The above objective is further achieved by a picture 
replacement device for replacing a specified area of a 
present picture with a desired picture comprising: pic- 
ture generation means for generating a desired picture 
for replacing; selection means for selecting a replace- 
ment area within a reference picture in advance and 
storing a position of the replacement area and a refer- 
ence picture within the replacement area; area detection 
means for selecting a predetermined number of areas 
within a present picture taken by a camera; calculating 
a difference between a present picture of each the se- 
lected area and the stored reference picture; and de- 
tecting an area presenting a minimum difference result 
as a replacement area; correction means for correcting 
a position of the stored replacement area based on an 
area position of the detected replacement area; key sig- 
nal generation means for generating a key signal indi- 
cating an Identical part of a present picture within the 
detected replacement area and a reference picture with- 
in a replacement area corrected by the correction 
means; and a switch selectable at a unit of one picture 
element pixel based on the key signal; responding to the 
key signal indicating an identical part of a present picture 
and a reference picture, the switch selects to input the 
desired picture and responding to the key signal indicat- 
ing a different part between a present picture and a ref- 
erence picture, the switch selects to input the present 
picture, 

The above objective is achieved by a picture re- 
placement method of replacing a specified area of a 
present picture with a desired picture comprising: gen- 
eration step of generating a desired picture for replac- 
ing; selection step of selecting a replacement area re- 
placed with a desired picture within reference picture 
that has been taken; storage step of storing a position 
of the replacement area and a reference picture within 
the replacement area; detection step of detecting an ar- 
ea as the replacement area within a present picture; key 
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signal generation step of generating a key signal indi- 
cating an identical part of a present picture and a refer- 
ence picture by comparing a present picture within the 
detected area and a reference picture within the stored 
replacement area; and replacement step of replacing s 
the identical part of a present picture and a reference 
picture with a part of the desired picture corresponding 
to the identical part based on the key signal. 

It is preferable that the present picture is taken in a 
condition using the same angle and direction as taking 
a reference picture. 

It is further preferable that the detection step further 
comprising: area detection step of selecting a predeter- 
mined number of areas expected to be a replacement 
area within a present picture; calculating a difference 
between a present picture of each the selected area and 
a reference picture that has been stored and detecting 
an area presenting a minimum difference result as a re- 
placement area; and correction step of correcting a shift 
between a position of an area delected by the area de- 
tection step and a position of a replacement area that 
has been stored in advance. 

It is preferable that the replacement step comprises 
a step of replacing an identical picture element pixel of 
a present picture. and a reference picture with a picture 
element pixel of a desired picture corresponding to the 
identical picture element pixel based on a key signal; 
and keeping a different picture element pixel between a 
present picture and a reference picture as picture ele- 
ment pixel of a present picture. 

This and other objects, features and advantages of 
the present invention will become more apparent upon 
a reading of the following detailed description and draw- 
ings, in which: 

Fig. 1 A to IE are views explaining a logo replace- 
ment method of the present invention; and 
Fig. 2 is a block diagram of a logo replacement sys- 
tem and a logo replacement method of the present 
invention. 

Prior to explaining the embodiment of the present 
invention in detail, the principle of the present invention 
is described referring to Fig. 1 A to Fig. 1 E. 

A reference picture is taken by a camera set at a 
basic camera position. Fig. 1 A shows the reference pic- 
ture taken by the camera. The reference picture is taken 
in the condition where nothing obstructs the picture with- 
in a replacement area described later. For example, the 
reference picture as shown in Fig. 1 A is taken so as not 
to cause any part of the picture OXAD to be masked by 
a person standing in front of the picture OXAD withiri 
the replacement area. 

A replacement area that is required to be replaced 
with a desired logo is selected within the reference pic- 
ture (Fig. 1 A). It is assumed in the present invention that 
a rectangle area enclosing Ox AD is specified as the re- 
placement area. The position of the replacement area 



to the reference picture and the picture within the re- 
placement area, i.e., OXAD are stored at a unit of one 
picture element pixel. 

After finishing the above-described preparation, the 
shooting is started. 

Upon start of shooting, a replacement area is de- 
tected within the picture taken by the camera (hereinaf- 
ter referred to as "present picture"). The detection is ex- 
ecuted in the following manner. 

An area at a position corresponding to the replace- 
ment area that has been stored in advance is assumed 
to be the replacement area. A difference between a 
present picture within the assumed replacement area 
and the stored reference picture is calculated. Then as 
shown in Fig. 1 B, the replacement area is shifted at a 
predetermined picture element pixel unit in 8 directions. 
Each difference between the present picture within each 
shifted area and the stored reference picture is calcu- 
lated, respectively. The area presenting the minimum 
difference value among the thus calculated differences 
is detected as the replacement area. As a result, the re- 
placement area can be detected accurately even when 
the shooting range of the camera shaken in the wind 
has been slightly shifted. 

A present picture within the detected replacement 
area and the stored reference picture are subjected to 
subtraction at a picture element pixel unit. When the pic- 
ture element pixel indicates an identical part of the 
present picture and the reference picture, the subtrac- 
tion result is 0. When the picture element pixel indicates 
a different part between the present picture and the ref- 
erence picture, the subtraction result is not 0. Based on 
the result, a key signal is generated. The key signal is 
used for identifying the picture element pixel indicating 
the identical part or different part of the above two pic- 
tures. When the key signal is 0, it normally indicates that 
the picture element pixel shows an identical picture ar- 
ea. When the key signal is 1 , it indicates that the picture 
element pixel shows a different picture area. Fig. 1C is 
a view showing a concept of the key signal. In Fig. 1C, 
the black area represents the picture element pixel 
where the present picture is identical to the reference 
picture. The white area represents the picture element 
pixel where the present picture is different from the ref- 
erence picture. As shown in Fig. 1C, the difference be- 
tween the present picture and the reference picture can 
be identified as above. More specifically, the area hid- 
den behind the person can be defined from the picture 
within the replacement area. 

Lastly the picture within the replacement area is re- 
placed with the logo using the key signal. It is assumed 
that the picture is replaced with the logo shown in Fig. 
1 D. The replacing logo has been generated in advance 
as well as other pictures within the replacement area. 
For example, the logo VDAX shown in Fig. 1 D and the 
periphery thereof are generated in pairs. When the key 
signal is 0 indicating the identical picture element pixel, 
the picture is replaced with the picture area of the gen- 
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erated logo corresponding to the identical picture ele- 
ment pixel. When the key signal is 1 indicating the dif- 
ferent picture element pixel, the present picture is kept 
unchanged. 

As shown in Fig. 1 E, the unseen area obstructed by 
persons is not replaced with the logo and the other area 
not obstructed by them is replaced with the logo so as 
to naturally show that the logo appears to have existed 
behind them. 

The system for realizing the aforementioned logo 
replacement method is described. 

Fig. 2 is a block diagram of the logo replacement 
system. 

The logo replacement system of the embodiment of 
the present invention comprises a camera 1 A and a logo 
replacement device 1 B. The logo replacement device 
1 B comprises a switch 2, a logo generation section 3, 
reference picture selection section 4, an area detection 
section 5, a key signal generation section 6 and a re- 
placement section 7. Each section of the above system 
Is described in detail. 

The camera 1A is used for TV broadcasting, having 
fixed direction and angle to a subject i.e.^ for shooting 
the specified range. 

The switch 2 inputs picture data taken by the cam- 
era 1 A for outputting to the reference picture selection 
section 4 or the area detection section 5, selectively. 
When the reference picture is selected before shooting, 
the picture data from the camera 1 A is output to the ref- 
erence picture selection section 4 At shooting, i.e., re- 
placing the logo, the picture data from the camera 1 A 
are output to the area detection section 5. 

The logo generation section 3 generates a desired 
logo that is required for replacement and stores the gen- 
erated logo. The logo is generated at a picture element 
pixel unit through computer graphics and the like. 
The reference picture selection section 4 comprises a 
picture storage circuit 41 , a replacement area selection 
circuit 42 and a replacement area storage circuit 43. 

The picture storage circuit 41 receives the picture 
data from the camera 1A. The input picture data is 
stored as a reference picture therein. 
The replacement area selection circuit 42 selects a po- 
sition of a desired replacement area within the stored 
reference picture. This processing is executed by dis- 
playing the reference picture on a monitor and then se- 
lecting the desired replacement area through a key- 
board, mouse or the like. 

The replacement area storage circuit 43 stores a 
position of the selected replacement area and the refer- 
ence picture within the replacement area. 

The area detection circuit 5 comprises a replace- 
ment area detection circuit 51 , a shift correction control 
circuit 52, a logo shift correction circuit 53 and a refer- 
ence picture shift correction circuit 54. 
The replacement area detection circuit 51 detects the 
replacement area within the present picture that has 
been televised. The replacement area detection circuit 
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51 assumes that the area locating at the position of the 
replacement area of the input present pictures that has 
been stored In advance Is the replacement area. Then 
a difference between the present picture within the as- 
5 sumed replacement area and the stored reference pic- 
ture is calculated. The replacement area is shifted per 
1/2 of the picture element pixel in 8 directions setting 
the assumed replacement area as a center. A difference 
between each present picture within the shifted replace- 

^0 ment areas and the stored reference picture data is cal- 
culated, respectively. The area presenting the minimum 
difference result is detected as the replacement area. 

The shift correction control circuit 52 controls the 
logo shift correction circuit 53 and the reference picture 

^5 shift correction circuit 54 responding to a number of pic- 
ture elements pixel shifted when detecting the replace- 
ment area sent from the replacement area detection cir- 
cuit 51 . The reference picture correction circuit 53 cor- 
rects the shift between the reference picture and the 

20 present picture based on the control signal from the shift 
correctbn control circuit 52. In the reference picture cor- 
rection circuit 53, the position of the present picture with- 
in the detected replacement area is corrected to coin- 
cide with the position of the reference picture within the 

25 replacement area that has been stored. 

The logo correction circuit 54 corrects a shift be- 
tween the logo and the present picture based on a con- 
trol signal from the shift correction control circuit 52. In 
the logo correction circuit 54, the present picture within 

30 the detected replacement area is corrected to coincide 
with the position of the logo that has been stored. 
The key signal generation section 6 comprises a sub- 
traction device 61 and a key signal generation circuit 62. 
The subtraction device 61 calculates a difference 

35 between the present picture data and the reference pic- 
ture with its position corrected by the reference picture 
correctbn circuit 53. 

The key signal generation circuit 62 generates a key sig- 
nal based on the subtraction result of the subtraction 

40 device 61 . The key signal generation circuit 62 gener- 
ates a key signal 0 when the result of the subtraction 
device 61 is 0 and a key signal 1 when the result of the 
subtraction device 61 is not 0. 

The replacement section 7 is a switch that switches 

4S at a unit of one picture element pixel based on the key 
signal. 

The replacement section 7 receives inputs of the 
logo from the logo correction circuit 54 and the present 
picture from the camera 1 A. When the key signal is 0, 

50 the replacement section 7 selects the logo for output- 
ting. When the key signal Is 1 , the replacement section 
7 selects the present picture for outputting. As a result, 
the present picture is obtained in which only a prodetcr- 
mined area is replaced with the logo. 

55 The logo replacement method is realized by the 
above-constructed system. 

The present Invention allows the logo to replace the 
picture within the replacement area naturally showing 
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that the logo appears to have been a part of the present 
picture. 

The unseen part of the replacement area of the present 
picture that has been hidden by the person or the object 
is not replaced by the logo and kept unchanged. The 
other part of the replacement area is replaced with the 
logo. 

Therefore replacement can be done so naturally as if 
the logo had existed in the present picture before. 

The entire disclosure of Japanese Patent Applica- 
tion No. 8-060449 filed on March 18, 1996 including 
specification, claims, drawing and summary are incor- 
porated herein by reference in its entirety. 



Claims 

1. A picture replacement system for replacing a spec- 
ified area of a present picture with a desired picture 
comprising; 

at least one camera; 

picture generation means for generating a de- 
sired picture for replacing; 
selection means for selecting a replacement ar- 
ea within a reference picture in advance and 
storing a position of said replacement area and 
a reference picture within said replacement ar- 
ea; 

detection means for detecting a replacement 
area within a present pictu re taken by said cam- 
era based on said position of said replacement 
area and said reference picture within said re- 
placement area; 

key signal generation means for generating a 
key signal indicating an identical part of a 
present picture within said detected replace- 
ment area and said stored reference picture; 
and 

replacement means for replacing an identical 
part of a present picture and a reference picture 
within a part picture of said desired picture cor- 
responding to said identical part based on said 
key signal. 

2. The picture replacement system of claim 1 , wherein 
said camera shoots a present picture at a fixed 
range lor shooting a reference picture. 

3. The picture replacement system of claim 1 , wherein 
said detection means comprises: 

area detection means for selecting a predeter- 
mined number of areas in a present picture, cal- 
culating a difference between a present picture 
within each said selected area and a stored ref- 
erence picture and detecting an area present- 
ing a minimum difference result as a replace- 



8 

ment area; and 

correction means for correcting a shift between 
said stored replacement area and said detect- 
ed replacement area based on an area position 
5 of said detected replacement area. 

4. The picture replacement system of claim 1 , wherein 
said replacement means comprises a switch se- 
lectable at a unit of one picture element pixel based 

10 on a key signal; responding to said key signal indi- 
cating an identical part of a present picture and a 
reference picture, said switch selects to input a part 
of desired picture corresponding to said identical 
part and responding to said key signal indicating a 
IS different part between a present picture and a ref- 
erence picture, said switch selects to input a part of 
present picture corresponding to said different part. 

5. A picture replacement system for replacing a spec- 
20 ifjed area of a present picture with a desired picture 

comprising: 

a camera for shooting a present picture at a 
fixed range for shooting a reference picture; 
2S picture generation means for generating a de- 

sired picture for replacing; 
selection means for selecting a replacement ar- 
ea within a reference picture in advance and 
storing a position of said replacement area and 
30 a reference picture within said replacement ar- 

ea; 

area detection means for selecting a predeter- 
mined number of areas within a present picture 
shot by said camera, calculating a difference 
35 between a present picture within each said se- 

lected area and said stored reference picture 
and detecting an area presenting a minimum 
difference result as a replacement area; 
correction means for correcting an area posi- 
40 tion of said shot replacement area based on an 

area position of said detected replacement ar- 
ea; 

key signal generation means for generating a 
key signal indicating an identical part of a 
45 present picture within said detected replace- 

ment area and said stored reference picture; 
and 

a switch selectable at a unit of one picture ele- 
ment pixel based on a key signal, for selecting 
50 to input said desired picture when said key sig- 

nal indicating an identical part of said present 
picture and said reference picture, for selecting 
input said present picture when said key signal 
indicating a different part between said present 
55 picture and said reference picture. 

6. A picture replacement device for replacing a spec- 
ified area of a present picture with a desired picture 
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comprising: 

picture generation means for generating a de- 
sired picture for replacing; 

selection means for selecting a replacement ar- s 
ea within a reference picture in advance and 
storing a position of said replacement area and 
a reference picture within said replacement ar- 
ea; 

area detection means for selecting a predeter- io 
mined number of areas within a present picture 
taken by a camera; calculating a difference be- 
tween a present picture of each said selected 
area and said stored reference picture; and de- 
tecting an area presenting a minimum differ- i5 
ence result as a replacement area; 
correction means for correcting a position of 
said stored replacement area based on an area 
position of said detected replacement area; 
key signal generation means for generating a 20 
key signal indicating an identical part of a 
present picture within said detected replace- 
ment area and a reference picture within a re- 
placement area corrected by said correction 
means; and 2S 
a switch selectable at a unit of one picture ele- 
ment pixel based on said key signal; respond- 
ing to said key signal indicating an identical part 
of a present picture and a reference picture, 
said switch selects to input said desired picture 30 
and responding to said key signal indicating a 
different part between a present picture and a 
reference picture, said switch selects to Input 
said present picture, 

35 

7. A picture replacement system for replacing a spec- 
ified area of a present picture with a desired picture 
comprising: 

picture generation means for generating a de- 40 
sired picture for replacing; 
a first storage circuit for storing a reference pic- 
ture; 

selection means for selecting a replacement ar- 
ea replaced with said desired picture within dis- 45 
played said reference picture; 
second storage means for storing a position of 
said selected replacement area within said ref- 
erence picture and a reference picture within 
said replacement area; 50 
area detection means for selecting a predeter- 
mined number of areas within a present picture 
taken by a camera, calculating a difference be- 
tween a present picture of each said selected 
area and said stored reference picture, and de- 55 
tecting an area presenting a minimum differ- 
ence result as a replacement area; 
correction means for correcting a position of 
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said replacement area stored In said second 
storage means and a position of said desired 
picture based on an area position of said de- 
tected replacement area; 
key signal generation means for generating a 
key signal Indicating a difference between a 
present picture and a reference picture based 
on a difference between a present picture with- 
in said detected replacement area and a refer- 
ence picture within said corrected replacement 
area; and 

a switch for inputting said present picture and 
said desired picture, outputting said desired 
picture when said key signal Indicates identical 
pictures thereof and outputting said present 
picture when said key signal indicates different 
pictures thereof. 

8. The picture replacement system of claim 7, wherein 
said area detection means comprises means for 
calculating a difference between a reference picture 
and a present picture at a picture element pixel unit 
and detecting an area presenting a minimum differ- 
ence result as a replacement area, 

9. The picture replacement system of claim 7, wherein 
said correction means comprises means for cor- 
recting a position at a unit of one picture element 
pixel. 

10. The picture replacement system of claim 7, wherein 
said key signal generation means comprises means 
for generating a key signal at a unit of one picture 
element pixel. 

11 . The picture replacement system of claim 7, wherein 
said switch is selectable at a unit of one picture el- 
ement pixel. 

12. A picture replacement method of replacing a spec- 
ified area of a present picture with a desired picture 
comprising: 

generation step of generating a desired picture 
for replacing; 

selection step of selecting a replacement area 
replaced with a desired picture within reference 
picture that has been taken; 
storage step of storing a position of said re- 
placement area and a reference picture within 
said replacement area; 

detection step of detecting an area as said re- 
placement area within a present picture; 
key signal generation step of generating a key 
signal indicating an identical part of a present 
picture and a reference picture by comparing a 
present picture within said detected area and a 
reference picture within said stored replace- 
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ment area; and 

replacement step of replacing said identical 
part of a present picture and a reference picture 
with a part of said desired picture correspond- 
ing to said identical part based on said key sig- s 
nal. 

13. The picture replacement method of claim 12, 
wherein said present picture is taken in a condition 
using the same angle and direction as taking a ref- to 
erence picture. 

14. The picture replacement method of claim 12, 
wherein said detection step further comprising: 

IS 

area detection step of selecting a predeter- 
mined number of areas expected to be a re- 
placement area within a present picture; calcu- 
lating a difference between a present picture of 
each said selected area and a reference picture 20 
that has been stored and detecting an area pre- 
senting a minimum difference result as a re- 
placement area; and 

correction step of correcting a shift between a 
position of an area dotocted by said area de- 2^ 
tection step and a position of a replacement ar- 
ea that has been stored in advance. 

15. The picture replacement method of claim 12. 
wherein said replacement step comprises a step of 30 
replacing an identical picture element pixel of a 
present picture and a reference picture with a pic- 
ture element pixel of a desired picture correspond- 
ing to said identical picture element pixel based on 

a key signal; and keeping a different picture element 35 
pixel between a present picture and a reference pic- 
ture as picture element pixel of a present picture. 
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